1. Field of the Invention
The present invention relates to a dynamic image signal coding method and apparatus, a dynamic image signal transmission method, and a signal recording medium which can preferably be applied when recording or reproducing a dynamic image signal to/from a recording medium as well as displaying the dynamic image on a display unit, and when transmitting a dynamic image signal from a transmission side through a transmission path to a reception side as in video conference system, a visual telephone system, a television broadcasting system, or the like.
2. Description of the Related Art
Conventionally, when a dynamic image signal is digitized to be recorded or transmitted, the dynamic image signal represents a great amount of data and the data is encoded (compressed). As a representative coding method, there is an coding method of motion compensative prediction type.
FIG. 1 shows a principle of the motion compensative prediction.
The motion compensative prediction is a coding system which utilizes a correlation in the temporal direction of an image signal. That is, a motion vector MV of a current image IO to be encoded is predicted from a motion prediction reference image IR which has been decoded. The image signal which is being decoded to be reproduced is moved in accordance with the signal (according to the motion vector MV), this motion information (motion vector MV) and a prediction residue at that time are transmitted, so as to compress a data amount required for coding.
The representative method of this motion compensative prediction is the so-called MPEG (moving picture expert group). This term represents a dynamic image coding method discussed and concluded by the working group WG11 of a sub committee SC29 of the Joint Technical Committee JTC1 of the ISO (International Standard Organization) and IEC (International Electric Conference).
According to the MPEG, an image (frame or field) is divided into macro blocks, each consisting of 16 lines .times.16 pixels, and the compensative prediction is carried out on this macro block basis. The motion compensative prediction can roughly be divided into an intra-coding method and a non-intra coding (inter-coding) method. The intra-coding method utilizes a data only on the macro block concerned, whereas the non-intra coding method utilizes both of a data on the macro block concerned and a data obtained from an image which appears at a different time.
According to the non-intra coding of the motion compensative prediction of the MPEG, it is necessary to detect a motion vector of a macro block of a motion prediction reference image corresponding to the current macro block of the image to be encoded and carry out a DCT (discrete cosine transform) to a difference between the predicted image according to this motion vector and an input image. The DCT coefficient obtained is digitized for carrying out a data compression.
Here, according to the MPEG, when detecting a motion vector of a macro block of a motion prediction reference image corresponding to the macro block of the image currently encoded, a pattern matching is carried out between the motion prediction reference image and the current image to be encoded on the macro block basis. More specifically, as shown in Equation (1) below, a motion vector is detected according to a value Ef which is a sum of absolute values of differences between a luminance signal Si,j of the macro block of the object image to be currently coded and a luminance signal Ri+mvx,j+mvy of the motion prediction reference image referenced by an arbitrary motion vector (mvx, mvy). EQU Ef=.SIGMA..vertline.S.sub.i,j -R.sub.i+mvx,j+mvy .vertline.(i=0 to 15, j=0 to 15) (1)
That is, the motion vector to be obtained is the (mvx, mvy) which makes the aforementioned Ef a minimum value.
Here, for example, if the image entered is affected by a noise or the like, the aforementioned motion vector detection may cause an erroneous detection of a motion vector where the image is flat, for example, or the detected motion vector may not be (0, 0) even when the image is actually still.
That is, when one or both of the aforementioned luminance signal Si,j and the luminance signal Ri+mvx,j+mvy which is to minimize the value of the Ef in the Equation (1) above is affected by a noise or the like, the value of the aforementioned Ef cannot become minimum. As a result, it is impossible to obtain a correct motion vector.
Especially when the image is still, and if the macro block which is expected to have a motion vector (0, 0) is affected by a noise or the like, as shown in FIG. 2, the motion vector does not become (0, 0) in spite of no motion present in the macro block. This significantly deteriorates the quality of the image. It should be noted that in the example of FIG. 2, among N frames F.sub.0 to F.sub.N-1, F.sub.i is the reference prediction image IR and the frame F.sub.i+j is the object image IO to be currently coded. Here, if the image is still, as the prediction reference image IR corresponding to the macro block MBo in the object image IO to be currently coded, a macro block MB.sub.RR of motion (0, 0) is to be detected so as to define the motion vector MV.sub.R as (0, 0). However, if a noise or the like is present, an error is caused in the pattern matching and an incorrect macro block MV.sub.RE is detected. In this case, a motion vector MV.sub.E of (mvx, mvy) different from the actual one is obtained. Thus, when the image is still, if a noise or the like is present, a motion vector MV.sub.E of (mvx, mvy) is obtained in stead of a motion vector MV.sub.R of (0, 0), deteriorating the image quality.
In order to solve this problem, when executing a motion compensation (hereinafter, referred to as MC) predicting a motion vector of a macro block, a predetermined decision is made using a luminance value of each macro block and according to the result of this decision, the motion vector is forcibly made to (0, 0). Hereinafter, this processing which forcibly makes the motion vector (0, 0) will be referred to as NonMC. Moreover, the aforementioned predetermined decision will be referred to as an MF/NonMC decision.
A so-called SM3 (Simulation Model 3) of the MPEG uses such a predetermined decision (MC/NonMC decision).
Description will now be directed to the MC/NonMC decision method and the motion compensation prediction method of the MPEG with reference to the attached drawings.
FIG. 3 shows an example of a moving image (dynamic image) signal coding apparatus which carries out the aforementioned MC/Non MC decision. The moving image signal encoding apparatus 41 shown in FIG. 3 is based on a hybrid coding method employing the motion comensative prediction and the DCT in combination.
In FIG. 3, an input moving image signal S41 inputted from a terminal 42 is transmitted to a scan converter 43. In the scan converter 43, an image signal inputted by raster scan is converted into a signal of a block format. That is, as shown in FIG. 4A, the image inputted by raster scan is a data formatted in a frame of H dots .times. V lines. The scan converter 43 divides the signal of one frame into N slices, each consisting of 16 lines. Each slice is divided into M macro blocks as shown in FIG. 4B. Each macro block consists of a luminance signals corresponding to 16.times.16 pixels (dots). The luminance signal is divided into blocks Y [1] to Y [4], each based on 8.times.8 dots. Moreover, this luminance signal of 16.times.16 dots is related to a color difference signal Cb [5] of 8.times.8 dots and a color difference signal Cr [6] of 8.times.8 dots.
The moving image signal S42 which has been converted by the scan converter 43 into a signal of the aforementioned format is transmitted to a motion vector prediction circuit 44, a NonMEerr calculation circuit 45, and a moving image coding circuit 48.
The motion vector prediction circuit 44 carries out a pattern matching between each of the macro blocks of the frames of the moving image signal S42 and a corresponding reference image of predetermined search range stored in a reference image storage block 46 (i.e., a pattern matching is carried out on a macro block basis between an image to be currently coded and a corresponding reference image). According to the results of this pattern matching, a motion vector MV is detected and simultaneously with this, a prediction residue MEerr is calculated, the result of which is outputted to an MC/NonMC decision circuit 47.
Here, if it is assumed that the macro block of the image to be currently coded has a luminance value Si,j and the motion prediction reference image has a luminance value Ri,j, the aforementioned motion prediction residue MEerr can be obtained from Equation (2) as follows. EQU MEerr=.SIGMA..vertline.S.sub.i,j -R.sub.i+mvx,j+mvy .vertline./256(i=0 to 15, j=0 to 15) (2)
When the moving image signal coding apparatus 41 has completed motion vector prediction of all the macro blocks contained in one frame, the current moving image signal S42 (i.e., image data of the frame) is stored in the reference image storage block 46, which is then used as a reference image for the next image to be inputted.
On the other hand, the NonMEerr calculation circuit 45 calculates a non-motion compensative residue NonMEerr for each of the macro blocks of the frames of the moving image signal S42 successively inputted. That is, the NonMEerr calculation circuit 45 calculates a difference between each of the macro blocks contained in the object image to be currently encoded and inputted frame after frame, and a macro block of motion vector (0, 0) among the reference images stored in the reference image storage block 46. The difference values obtained are outputted for each of the macro blocks as a non-motion compensative residue NonMEerr. Here, the aforementioned non-motion compensative residue NonMEerr is obtained by Equation (3) as follows. EQU NonMEerr=.SIGMA..vertline.S.sub.i,j -R.sub.i,j .vertline./256 (i=0 to 15,j=0 to 15) (3)
The non-motion compensative residue NonMEerr thus obtained is transmitted to the MC/NonMC decision circuit 47.
The MC/NonMC decision circuit 47 is supplied with the aforementioned motion prediction residue MEerr and the non-motion compensative residue NonMEerr and compares the motion prediction residue MEerr to the non-motion compensative residue NonMeerr by using a region division as shown in FIG. 5 for example, so as to make a MC/NonMC decision. That is, in FIG. 5, the vertical axis represents the residue NonMEerr and the horizontal axis represents the motion prediction residue MEerr. The MC/NonMC decision circuit 47 makes the MC/NonMC decision not according to the comparison regions separated by the straight line of NonMEerr=MEerr indicated by a dotted line in the figure but according the comparison regions separated by the solid line indicated in the figure consisting of a straight line connecting (0, 1) to (0, 5, 1), a straight line connecting 90.5, 1) to (1, 1.5), a straight line connecting (1, 1.5) to (1, 2.5), a straight line connection (1, 2.5) to (1.5, 3), a straight line connecting (1.5, 3) to (2.7, 3), and a straight line of NonMEerr=0.9 MEerr+0.57. When the aforementioned motion prediction residue MEerr value is compared to the non-motion compensative residue NonMEerr value, a decision is made to be MC if the value is in the upper left to the aforementioned solid line and NonMC if the value is in the lower right to the solid line.
When the aforementioned MC/NonMC decision circuit 47 made a decision of MC, the motion vector MV (mvx, mvy) inputted is outputted as it is; and when a decision of NonMC is made, the motion vector MV (mvx, mvy) inputted is converted into (0, 0) for output. The motion vector MV outputted from this MC/NonMC decision circuit 47 is supplied to the moving image coding circuit (hybrid encoder) 48 together with the moving image signal S42 of the frame.
Subsequently, FIG. 6 shows a specific configuration of the moving image coding circuit (hybrid encoder) 48 of Fig. FIG. 3.
In this FIG. 6, an input terminal 61 is supplied with the motion vector MV from the aforementioned MC/NonMC decision circuit 47. On the other hand, an input terminal 50 is supplied with an input moving image signal S60 to this moving image coding circuit 48. It should be noted that the input moving image signal S60 corresponds to the moving image signal S42 supplied from the scan converter 43 in FIG. 3.
Moreover, the moving image coding circuit 48 includes a motion compensation inter/intra frame prediction circuit 57 which has an image memory and outputs a prediction image signal S63 which has been read from this image memory according to the motion vector MV from the input terminal 61.
A calculator 51 carries out an addition by using the input moving image signal S60 (S42) from the input terminal 50 as an addition signal and the prediction image signal S63 from the motion compensation inter/intra frame prediction circuit 57 as a subtraction signal, thus calculating a difference between the input moving image signal S42 and the prediction image signal S63, the result of which calculation is outputted as a prediction residue signal S61. It should be noted that the aforementioned prediction is not carried out when a signal having no correlation with an adjacent frame such as a scene change signal is supplied as an input moving image signal. In this case, the input moving image signal S 60 is taken out of the calculator 51 as it is (outputted as an original signal without executing prediction).
Next, the prediction residue signal S61 (or original signal if no prediction is executed) outputted from the calculator 51 is supplied to a DCT circuit 52. This DCT circuit carries out a two-dimensional DCT to the aforementioned prediction residue signal S61. A DCT coefficient outputted from this DCT circuit 52 is converted into a scalar quantity by a quantization circuit 53. The quantization output signal from this quantization circuit 53 is transmitted to a variable length coding (VLC) circuit 58 and to a inverse quantization circuit 54. The VLC circuit 58 carries out, for example, a Huffman coding to the aforementioned quantization output signal. An output signal from this VLC circuit 58 is supplied to a buffer memory 59. The buffer memory 59 smoothes the bit rate for output from an output terminal 60 to a transmission path. When this buffer memory 59 is almost overflown, this is reported as a quantization control information back to the quantization circuit 53. The quantization circuit 53 increases a quantization step so as to reduce a data amount outputted from the quantization circuit 53, thus preventing overflow of the buffer memory 59. Note that the output terminal 60 in FIG. 6 corresponds to the output terminal 49 in FIG. 3.
On the other hand, the inverse quantization circuit 54 carries out an inverse quantization to the aforementioned quantization output signal in accordance with a quantization step information q.sub.-- step supplied from the quantization circuit 53. An output signal from this inverse quantization circuit 54 is supplied to an inverse DCT circuit 55, where the signal is subjected to the inverse DCT processing. The prediction residue signal S62 which has been decoded is supplied to a calculator 56.
This calculator 56 is also supplied with a signal identical to the prediction image signal S63 which is supplied to the calculator 51. The calculator 56 adds the prediction image signal S68 to the aforementioned prediction residue signal S63, thus enabling to obtain an image signal which is locally decoded. This image signal is identical to the output image at the reception side.
The SM3 of the MPEG thus makes the MC/NonMC decision for carrying out moving image coding. Here, if the decision is made to be NonMC, there is no need of transmitting a motion vector. As shown in FIG. 5, for those areas where the prediction error signal is near to 0 and the areas where the motion prediction residue MEerr value is almost identical to the non-motion compensative residue NonMEerr or the prediction residue MEerr is small, it is possible to reduce the data amount of the motion vector by selecting NonMC.
Next, when carrying out a motion compensative coding, an image inputted may have no motion in the macro block between the object image to be currently coded and the motion prediction reference image but have a great change in the luminance value. For example, such an image may be a still image which has been subjected to a fade-in or fade-out processing for gradually changing the luminance value. It should be noted that in the explanation below, such an image which has been subjected to a fade-in or fade-out processing will be referred to as a fade image.
When an input image in which the luminance value is changed such as in the fade image is subjected to a motion compensative prediction coding by way of pattern matching, the luminance value is greatly changed between the macro blocks corresponding to the object image to be currently coded and the prediction reference image. Consequently, even if the macro block is to have a motion vector (0, 0), the aforementioned Ef is difficult to become minimum, causing an incorrect detection of a motion vector, which results in a block distortion and a mosquito noise in the reproduced image, thus deteriorating the image quality.
That is, for example, in the SM3 of the MPEG, the sum of the absolute value of the difference in the luminance values alone is used for the motion vector detection and the MC/NonMC decision. Accordingly, even when no motion is present between the macro block of the motion prediction reference image and the macro block of the object image to be currently coded, there is a possibility that the motion vector (0, 0) cannot be correctly detected because of a large change in the luminance value between these macro blocks and a macro block different from the essential motion may be detected as a macro block of the reference image. This lowers the coding efficiency and deteriorates the subjective image quality.
To cope with this, the applicant of the present invention disclosed in Japanese Patent Laid-Open Hei 8-98187 (Apr. 12, 1996) a method of obtaining a prediction residue signal of a macro block in case of NonMC by using, instead of the sum of absolute values of a difference in luminance values, a sum of absolute values of differences in AC (alternative current) components of the luminance value in the macro block, for comparison with the sum of absolute values of the luminance values in case of motion compensation, for carrying out the MC/NonMC decision (hereinafter, this method will be referred to as ACME). Note that corresponding to this Japanese Patent application, there is a U.S. Application Ser. No. 08/507/103 (Filed date: Jul. 25, 1995).
FIG. 7 shows a configuration example of the moving image signal coding apparatus using the aforementioned ACME. This apparatus includes an ECMEerr calculation circuit 25 and an MC/NonMC decision circuit 27. Moreover, this moving image signal coding apparatus includes an input terminal 22, a scan converter 23, a motion vector prediction circuit 24, a reference image storage block 26, a moving image coding circuit 28, and an output terminal 29 which respectively correspond to the input terminal 42, the scan converter 43, the motion vector prediction circuit 44, the reference image storage block 46, the moving image coding circuit 48, and the output terminal 49 of FIG. 3, operating identically and accordingly, their explanations will be omitted. Moreover, the signal S22 in FIG. 7 corresponds to the input moving image signal S60 in the moving image coding circuit 28 (the moving image signal S42 in FIG. 3).
In the ACMEerr calculation circuit of FIG. 7, if ACMEerr is assumed to be a sum of absolute values of the difference between the macro block of the object image to be currently coded and the macro block of the reference image of the motion vector (0, 0), then the ACMEerr can be obtained by Equation (4) as follows. EQU ACMEerr=.SIGMA..vertline.S.sub.i,j -S.sub.av)-(R.sub.i,j -R0.sub.av).vertline.(i=0 to 15, j=0 to 15) (4)
In this Equation (4), S.sub.av represents an average value of the luminance values in the macro block of the object image to be currently coded and R0.sub.av represents an average value of luminance values of the macro block of motion vector (0, 0) of the prediction reference image, which are respectively obtained by Equations (5) and (6) below. EQU S.sub.av =.SIGMA..vertline.S.sub.i,j /256(i=0 to 15, j=0 to 15)(5) EQU R0.sub.av =.SIGMA..vertline.R.sub.i,j .vertline./256(i=0 to 15, j=0 to 15)(6)
Moreover, in the MC/NonMC decision circuit 27, a decision is made by using a region division as shown in FIG. 8 for example. That is, in FIG. 8, the vertical axis represents the aforementioned ACMEerr, and the horizontal axis represents the aforementioned motion prediction residue MEerr. In this MC/NonMC decision circuit 27, the MC/NonMC decision is made not according to the comparison regions separated by the straight line of ACMEerr=MEerr indicated by a dotted line in the figure, but according to the comparison regions separated by a solid line in the figure consisting of a straight line connecting (0, y1) to (x1, y1), a straight line connecting (x1, y1) to (x2, y2), and a straight line of ACMEerr=MEerr+(x2 -y2). If the comparison between the MEerr and the ACMEerr results in the upper left of the aforementioned solid line, then the decision is made to be MC, and if in the lower right of the aforementioned solid line, the decision is made to be NonMC. It should be noted that in this FIG. 8, x1.gtoreq.y1 and x2 &gt;y2. The MC/NonMC decision circuit 27, in this region separation shown in FIG. 8, forcibly makes a decision of NonMC in an area where the ACMEerr and the MEerr are both near to 0 and makes a decision of MC in an area where the MCMEerr and the MEerr has almost identical values or the ACMEerr is greater than the MEerr.
In the moving image coding apparatus 21 shown in FIG. 7, the aforementioned decision is made so that a motion vector of (0, 0) can be correctly obtained even when the luminance greatly changes between the macro blocks corresponding to the object image to be currently coded and the motion prediction reference image and no motion is present such as in a fade image.
It is now assumed that a small residue is obtained for the portion having a flat input image when the MEerr and the NonMEerr are calculated between the object image to be currently coded and the motion prediction reference image by using the aforementioned Equations (2) and (3).
The aforementioned MC/NonMC decision of the SM3 of the MPEG cannot determine whether these residues are caused by a change of a small luminance change between the macro blocks or a noise or because a part of an object in the image is contained in the block. That is, when the macro blocks corresponding to the object image to be currently coded and the prediction reference image are both flat and the motion prediction residue is small, the MC/NonMC decision of SM3 of the MPEG cannot distinguish whether the macro block of the object image to be coded is a macro block containing a part of the image having a motion or a macro block containing a noise having no motion.
This will be detailed with reference to FIG. 9. FIG. 9A shows a part of an image of one frame containing an image of an object OB.sub.B which moves with lapse of time. The frame image shown in this FIG. 9A consists of flat macro blocks having almost no luminance change (i.e., macro blocks having only a slight change of the luminance value) excluding the macro blocks including the aforementioned object OB.sub.B, but the macro block MBFn contains a noise n. It is assumed that the macro block MB.sub.Fn containing this noise n is also a flat macro block having almost no change in the luminance value.
In a case when the MC/NonMC decision of SM3 of the MPEG is used for coding the frame image shown in this FIG. 9A so that the NonMC is selected in an area having a small residue, it is possible to make (0, 0) the motion vector of the macro block MB.sub.Fn having no motion but containing the noise n, but there is a danger that a flat block having only a slight change in the luminance value such as a macro block MB.sub.FI containing a part ob.sub.B of the image of the aforementioned object OB.sub.B for example may be decided to be a macro block having no motion (a macro block with a motion vector (0, 0)).
That is, if a macro block having a motion is identified as a block having no motion (with motion vector (0, 0)), as shown in FIG. 9B, the part ob.sub.B of the aforementioned object OB.sub.B remains as a fixed noise ob.sub.A in the reproduction image after a lapse of time t from the frame image of FIG. 9A. Note that FIG. 9B shows the OB.sub.A after the lapse of time t together with a position of the object OB.sub.B indicated by a dotted line which existed prior to the laps of time t.
In order to prevent the part ob.sub.B of the object OB.sub.B from remaining as the fixed noise ob.sub.A, one solution is that, for example, the NonMC is made to be easily selected in an area having a small motion prediction residue. However, in the case of the aforementioned conventional MC/NonMC method, it is not yet realized to make the NonMC be easily selected in the aforementioned area having a small prediction residue.
Moreover, in the case of the conventional MC/NonMC method, a motion vector is incorrectly detected, for example, in a macro block such as the aforementioned macro block MB.sub.Fn which is affected by the noise n, and which is flat, has no motion, and has a small prediction residue, resulting in deterioration of the coding efficiency.
These problems still remain in the MC/NonMC decision method using the aforementioned ACME disclosed in the aforementioned Japanese Patent Laid-Open Hei 8-98187.